Skip to main content

How to build an AR/AI iOS app with virtual fitness coach using QuickPose

Introduction​

QuickPose SDK allows you to build an AR/AI Fitness app with a virtual fitness coach in just one day. In this guide, we will show you how to build a simple app that enables two exercises: bicep curls and squats. The app will utilize QuickPose SDK to track the user's pose and provide feedback. Additionally, it will utilize Apple's Text-to-Speech SDK to deliver voice instructions to the user.

In order to understand basic concepts of QuickPose SDK, please follow the steps in QuickPose Integration Guide. In this guide, our aim is to show you the best practices for building a fitness app with QuickPose SDK.

Project structure overview​

Our project will follow the following workflow:

  1. The user selects an exercise.
  2. The user specifies the desired number of reps or the duration for the exercise.
  3. The user is prompted to turn up the volume for voice commands.
  4. Instructions are provided to the user on how to position the phone on the floor against the wall.
  5. Bounding Box feedback is provided to the user, indicating that they should remain within the box for 1-2 seconds.
  6. The exercise begins, and the user receives feedback on their form and the remaining number of reps or time.
  7. Once the exercise is completed, the user is presented with the results, which are saved to the local JSON file.

Selecting an exercise​

To define the available exercises, we will utilize the list of QuickPose features. Each exercise will utilize the .fitness feature to represent the exercise itself, and the .overlay feature to display the corresponding human pose. It is possible to employ different overlays for different exercises, apply conditional styling, or highlight specific sets of joints. For more information and detailed instructions, please refer to our Annotation and Styling Guide.

struct Exercise: Identifiable, Hashable {
    let id = UUID()
    let name: String
    let details: String
    let features: [QuickPose.Feature]
    // Add more properties as needed
}

let exercises = [
    Exercise(
        name: "Biceps Curls",
        details: "Lift weights in both hands by bending your elbow and lifting them towards your shoulder.",
        features: [.fitness(.bicepsCurls), .overlay(.upperBody)]
    ),
    Exercise(
        name: "Squats",
        details: "Bend your knees and lower your body.",
        features: [.fitness(.squats), .overlay(.wholeBody)]
    ),

]

QuickPoseBasicView — main exercise view​

The QuickPoseBasicView serves as the primary view of the app and is responsible for the following functionalities:

  • Showing the live video feed from the camera.
  • Providing visual and audio feedback to the user.
  • Displaying the number of reps and the remaining time.

We suggest utilizing states to manage the workout flow. There are seven states used for the exercise:

enum ViewState: Equatable {
    case startVolume
    case instructions
    case introBoundingBox
    case boundingBox
    case introExercise(Exercise)
    case exercise(SessionData, enterTime: Date)
    case results(SessionData)
}

startVolume and instructions states​

These states are used to ask a user to turn on the volume and place the phone on the floor against the wall. These states are used only once per a session.

Condition to switch to the next introBoundingBox state: when the user taps on "Start Workout" button.

introBoundingBox and boundingBox states​

These two states are used to ask a user to stand inside the bounding box and wait for 2 seconds. introBoundingBox state means that user is not in the bounding box. boundingBox state means that user is inside the bounding box.

There are two ways how this can be implemented:

  1. Use QuickPose inside feature. It will draw a box for the user to enter. By default, it is green when the user is inside the box, and red when the user is outside the box. You can change the colors and the size of the box.
  2. Use custom implementation in order to have more control over the UI. For example, you can do the following animation (see the code here) in order to show the user that they are inside the bounding box:

Condition to switch to the next introExercise state: when the user is inside the bounding box for 2 seconds.

introExercise state​

introExercise state is used to introduce the exercise.

Introduction state

During this state, you may consider showing exercise video, or giving some instructions to the user.

exercise state​

During exercise state we track the user's pose, give feedback, and count exercise repetitions. All of this is done using QuickPose SDK.

case .exercise(_, let enterDate):
let secondsElapsed = Int(-enterDate.timeIntervalSinceNow)

if let feedback = feedback[.fitness(.bicepCurls)] {
    feedbackText = feedback.displayString
} else {
    feedbackText = nil
    
    if case .fitness = sessionConfig.exercise.features.first, let result = features[sessionConfig.exercise.features.first!] {
        _ = counter.count(result.value) { newState in
            if !newState.isEntered {
                Text2Speech(text: "\(counter.state.count)").say()
                DispatchQueue.main.asyncAfter(deadline: .now()+0.1) {
                    withAnimation(.easeInOut(duration: 0.1)) {
                        countScale = 2.0
                    }
                    DispatchQueue.main.asyncAfter(deadline: .now()+0.4) {
                        withAnimation(.easeInOut(duration: 0.2)) {
                            countScale = 1.0
                        }
                    }
                }
            }
        }
    }
}

results state​

When specific condition like time or number of reps is met, we stop QuickPose and switch to the results state. In this state, we show the results of the exercise.

var hasFinished = false
if sessionConfig.useReps {
    hasFinished = counter.state.count >= sessionConfig.nReps
} else {
    hasFinished = secondsElapsed >= sessionConfig.nSeconds + sessionConfig.nMinutes * 60
}

if hasFinished {
    state = .results(newResults)
    quickPose.stop()
}

Voice commands synthesis​

During the workout, it can be valuable to give users feedback and instructions. Because usually they are far from the camera, it makes sense to use voice commands. In our example, we show how to use Apple's SDK to synthesize voice commands.

import Foundation
import AVFoundation


class Text2Speech {
    static let synthesizer = AVSpeechSynthesizer()
    
    var isSaid = false
    let text: String
    
    init(text: String) {
        self.text = text
    }
    
    func say() {
        if (isSaid) {
            return
        }
        
        let utterance = AVSpeechUtterance(string: self.text)
        Text2Speech.synthesizer.speak(utterance)
        
        self.isSaid = true
    }
}

This implementation allows to make sure that each phrase is said only once. In order to define feedback messages, check our feedback feature.

Conclusion​

The full code for FitCount app is available on GitHub. Feel free to open GitHub issues in case of any questions or suggestions.

You can also try the TestFlight version of the FitCount app here.

Start building your own fitness app with QuickPose SDK today! Sign up for a free account and get started with QuickPose SDK.